[create-vsix] Create `.vsix` files #541

jonpryor · 2017-03-30T02:37:34Z

Visual Studio 2017 uses .vsix files for Xamarin.Android SDK support.
A .vsix file is ZIP container with additional metadata, and the
Microsoft.VSSDK.BuildTools NuGet package contains various MSBuild
targets and tools to assist in creating .vsix files.

Add a new build-tools/create-vsix/create-vsix.csproj project to
create the bin/Build$(Configuration)/Xamarin.Android.Sdk*.vsix file,
so that we can plausibly provide per-build OSS Xamarin.Android
releases that work with Visual Studio 2017.

Unfortunately those tools were written on Windows, and not really well
tested on macOS or Linux... In particular, there are case-sensitivity
and directory-separator-char issues with the tooling, necessitating
that MONO_IOMAP=all be exported in order to run them:

    MONO_IOMAP=all MONO_OPTIONS=--arch=64 msbuild \
            build-tools/create-vsix/create-vsix.csproj /p:CreateVsixContainer=True

Meanwhile, we're still attempting to allow things to be built with
xbuild ¹. Thread this needle by "special-casing" the
$(BuildDependsOn), $(CopyVsixManifestFileDependsOn), and
$(DetokenizeVsixManifestFileDependsOn) MSBuild properties so that
when the $(CreateVsixContainer) MSBuild property is False -- the
default -- no .vsix package will be created, and xbuild will be
able to build the project.

Similar needle threading is needed to "support" building on Linux: the
Microsoft.VSSDK.BuildTools NuGet package uses case in an inconsistent
fashion -- or is it nuget? -- which results in
failures when building on Linux because nuget extracts e.g.:

    packages/Microsoft.VSSDK.BuildTools.15.0.26201/tools/vssdk/Microsoft.VsSDK.targets

while Microsoft.VSSDK.BuildTools attempts to <Import/> the file:

    packages/Microsoft.VSSDK.BuildTools.15.0.26201/tools/VSSDK/Microsoft.VsSDK.targets

vssdk != VSSDK on case-sensitive filesystems, so this results in
an error on Ubuntu (and presumably case-senstive macOS as well).

Handle the Linux case by extending the xbuild case: even when
$(CreateVsixContainer) is True, we won't actually build the .vsix
unless the $(VsSDKInstall) directory exists, which is the
tools/VSSDK path.

Building with $(CreateVsixContainer) set to True will require using
msbuild with MONO_IOMAP=all and MONO_OPPTIONS=--arch=64
exported, while using a case-insensitive filesystem.

The new make create-vsix CONFIGURATIONS=Debug target can be used to
explicitly create the Xamarin.Android.Sdk*.vsix file.

One problem with creating .vsix files: macOS still defaults to using
a 32-bit process for mono, and make create-vsix regularly fails
for me with an OutOfMemoryException when attempting to generate a
.vsix file ~600-700+MB in size. (Why so large? In part because for
Debug configuration we're including un-strip'd native libraries;
libmonosgen-2.0.dll is 237MB in size (!); it's closer to 5MB when
strip'd, but any un-strip'd MXE-generated native libraries to the
.vsix file quickly results in OutOfMemoryExceptions.)

A 64-bit mono can be used by using mono64 -- which isn't easily done
with msbuild -- or by using mono --arch=64, which can be done
with msbuild by exporting MONO_OPTIONS=--arch=64.
The make create-vsix target does this.

Finally, once we have a .visx being created, we then examine the
contents of the .vsix file, which is...weird:

    $MSBuild/Xamarin/Android/Xamarin.Android.CSharp.targets                                           # Desirable
    $ReferenceAssemblies/Microsoft/Framework/MonoAndroid/v1.0/Xamarin.Android.CSharp.targets          # wat?
    $ReferenceAssemblies/Microsoft/Framework/MonoAndroid/v1.0/Facades/Xamarin.Android.CSharp.targets  # really?!
    $ReferenceAssemblies/Microsoft/Framework/MonoAndroid/v7.1/Xamarin.Android.CSharp.targets          # !!!!

Turns Out™, the problem was due to @(ProjectReference). We're
(ab)using @(ProjectReference) to explicitly specify project
dependencies, and the (implicit) project build order.

Unexpectedly (forgetten?), MSBuild also copies the outputs of
@(ProjectReference)s into the $(OutputPath) of the current
project.

Since e.g. Mono.Posix.csproj had a reference on
Xamarin.Android.Build.Tasks.csproj, the result of this is that the
$(OutputPath) of Mono.Posix.csproj --
$prefix/lib/xbuild-frameworks/MonoAndroid/v1.0 -- also contained the
%(None.CopyToOutputDirectory) items from
Xamarin.Android.Build.Tasks.csproj.

Oops.

The fix is to set %(ProjectReference.Private) to False, which
disables this copying of additional and undesirable files.

But for how much longer? ↩

jonpryor · 2017-03-30T02:38:49Z

TODO: Double-check how the XamarinVS repo creates the .vsix file, and add the Windows-specific MSBuild targets.

kzu · 2017-03-30T14:45:51Z

build-tools/create-vsix/source.extension.vsixmanifest

+	<Installation AllUsers="true">
+		<InstallationTarget Version="[15.0,)" Id="Microsoft.VisualStudio.Community" />
+	</Installation>
+	<Dependencies>


Let's remove this node altogether. It's not needed.

kzu · 2017-03-30T14:47:04Z

build-tools/create-vsix/source.extension.vsixmanifest

+<?xml version="1.0" encoding="utf-8"?>
+<PackageManifest Version="2.0.0" xmlns="http://schemas.microsoft.com/developer/vsx-schema/2011" xmlns:d="http://schemas.microsoft.com/developer/vsx-schema-design/2011">
+	<Metadata>
+		<Identity Id="4489ebd9-10e4-42d0-bc3c-0639c608aaea" Version="1.0" Language="en-US" Publisher="Company" />


Let's make these match the values in https://github.com/xamarin/XamarinVS/blob/master/src/Installer/Android.Sdk/source.extension.vsixmanifest#L4

Note how the version can be fetched dynamically by invoking an MSBuild target that can read it from wherever: |Android.Sdk;GetVsixVersion| (that's |PROJECT_NAME;TARGET_NAME|)

kzu · 2017-03-30T19:32:40Z

build-tools/create-vsix/create-vsix.csproj

+    <_BuildVsix Condition=" '$(_BuildVsix)' == '' ">False</_BuildVsix>
+  </PropertyGroup>
+  <Import Project="..\..\Configuration.props" />
+  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">


Given that no code will ever run from the VSIX itself, how about just removing entirely the two property groups that are conditioned and just add one property below _BuildVsix as <OutputPath>bin\$(Configuration)</OutputPath> and just declare the two conditioned property groups empty just to satisfy VS?

<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' " /> <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' " />

Historically, Xamarin Studio gets rather cross if It doesn't have an explicit Debug property group and Release property group. I haven't tried in this case, but odds are against me.

kzu · 2017-03-30T19:33:24Z

build-tools/create-vsix/create-vsix.csproj

+    <WarningLevel>4</WarningLevel>
+    <ConsolePause>false</ConsolePause>
+  </PropertyGroup>
+  <ItemGroup>


This ItemGroup (and the AssemblyInfo.cs itself) shouldn't be needed if we aren't including any built outputs?

kzu · 2017-03-30T19:44:27Z

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Sdk.props

+-->
+<Project InitialTargets="RedirectMonoAndroidSdkPaths;RemoveSdksCache" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+
+	<PropertyGroup>


This whole file was intended only to work on VS2017+, which is true as long as '$(VsInstallRoot)' != ''. Maybe the property groups and targets in this file should have the condition, just to make it more explicit and avoid unintentional usage outside of that scenario?

Visual Studio 2017 uses `.vsix` files for Xamarin.Android SDK support. A `.vsix` file is [ZIP container with additional metadata][0], and the [Microsoft.VSSDK.BuildTools NuGet package][1] contains various MSBuild targets and tools to assist in creating `.vsix` files. Add a new `build-tools/create-vsix/create-vsix.csproj` project to create the `bin/Build$(Configuration)/Xamarin.Android.Sdk*.vsix` file, so that we can plausibly provide per-build OSS Xamarin.Android releases that work with Visual Studio 2017. Unfortunately those tools were written on Windows, and not really well tested on macOS or Linux... In particular, there are case-sensitivity and directory-separator-char issues with the tooling, necessitating that `MONO_IOMAP=all` be exported in order to run them: MONO_IOMAP=all MONO_OPTIONS=--arch=64 msbuild \ build-tools/create-vsix/create-vsix.csproj /p:CreateVsixContainer=True Meanwhile, we're still attempting to allow things to be built with `xbuild` [^3]. Thread this needle by "special-casing" the `$(BuildDependsOn)`, `$(CopyVsixManifestFileDependsOn)`, and `$(DetokenizeVsixManifestFileDependsOn)` MSBuild properties so that when the `$(CreateVsixContainer)` MSBuild property is False -- the default -- no `.vsix` package will be created, and `xbuild` will be able to build the project. Similar needle threading is needed to "support" building on Linux: the Microsoft.VSSDK.BuildTools NuGet package uses case in an inconsistent fashion -- or is it `nuget`? -- which results in [failures when building on Linux][2] because `nuget` extracts e.g.: packages/Microsoft.VSSDK.BuildTools.15.0.26201/tools/vssdk/Microsoft.VsSDK.targets while Microsoft.VSSDK.BuildTools attempts to `<Import/>` the file: packages/Microsoft.VSSDK.BuildTools.15.0.26201/tools/VSSDK/Microsoft.VsSDK.targets `vssdk` != `VSSDK` on case-sensitive filesystems, so this results in an error on Ubuntu (and presumably case-senstive macOS as well). Handle the Linux case by extending the `xbuild` case: *even when* `$(CreateVsixContainer)` is True, we won't actually build the `.vsix` unless the `$(VsSDKInstall)` directory exists, which is the `tools/VSSDK` path. Building with `$(CreateVsixContainer)` set to True will require using `msbuild` with `MONO_IOMAP=all` and `MONO_OPPTIONS=--arch=64` exported, while using a case-insensitive filesystem. The new `make create-vsix CONFIGURATIONS=Debug` target can be used to explicitly create the `Xamarin.Android.Sdk*.vsix` file. One problem with creating `.vsix` files: macOS still defaults to using a 32-bit process for `mono`, and `make create-vsix` regularly fails for me with an `OutOfMemoryException` when attempting to generate a `.vsix` file ~600-700+MB in size. (Why so large? In part because for Debug configuration we're including un-`strip`'d native libraries; `libmonosgen-2.0.dll` is *237MB* in size (!); it's closer to 5MB when `strip`'d, but any un-`strip`'d MXE-generated native libraries to the `.vsix` file quickly results in `OutOfMemoryException`s.) A 64-bit mono can be used by using `mono64` -- which isn't easily done with `msbuild` -- or by using `mono --arch=64`, which *can* be done with `msbuild` by exporting `MONO_OPTIONS=--arch=64`. The `make create-vsix` target does this. Finally, once we have a `.visx` being created, we then examine the *contents* of the `.vsix` file, which is...weird: $MSBuild/Xamarin/Android/Xamarin.Android.CSharp.targets # Desirable $ReferenceAssemblies/Microsoft/Framework/MonoAndroid/v1.0/Xamarin.Android.CSharp.targets # wat? $ReferenceAssemblies/Microsoft/Framework/MonoAndroid/v1.0/Facades/Xamarin.Android.CSharp.targets # really?! $ReferenceAssemblies/Microsoft/Framework/MonoAndroid/v7.1/Xamarin.Android.CSharp.targets # !!!! Turns Out™, the problem was due to `@(ProjectReference)`. We're (ab)using `@(ProjectReference)` to explicitly specify project dependencies, and the (implicit) project build order. Unexpectedly (forgetten?), MSBuild *also* copies the *outputs* of `@(ProjectReference)`s into the `$(OutputPath)` of the current project. Since e.g. `Mono.Posix.csproj` had a reference on `Xamarin.Android.Build.Tasks.csproj`, the result of this is that the `$(OutputPath)` of `Mono.Posix.csproj` -- `$prefix/lib/xbuild-frameworks/MonoAndroid/v1.0` -- also contained the `%(None.CopyToOutputDirectory)` items from `Xamarin.Android.Build.Tasks.csproj`. Oops. The fix is to set `%(ProjectReference.Private)` to False, which disables this copying of additional and undesirable files. [0]: https://msdn.microsoft.com/en-us/library/dd997148.aspx [1]: https://www.nuget.org/packages/Microsoft.VSSDK.BuildTools/ [2]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-anroid-linux-pr-builder/297/consoleText [^3]: But for how much longer?

Changes: dotnet/java-interop@f3553f4...d61b329 * dotnet/java-interop@d61b329 [Java.Interop-Tests] Relax Exception.Message checks (dotnet#544) * dotnet/java-interop@d778204 [Java.Interop] Better linker-friendly JavaObjectArray<T>.ValueMarshaler (dotnet#546) * dotnet/java-interop@4bd07e0 [vs-code] Recommend the zbecknell.t4-support extension (dotnet#545) * dotnet/java-interop@3b24a2c [Java.Interop] apply AggressiveInlining where profiler shows calls (dotnet#541)

Changes: dotnet/java-interop@f3553f4...d61b329 * dotnet/java-interop@d61b329 [Java.Interop-Tests] Relax Exception.Message checks (#544) * dotnet/java-interop@d778204 [Java.Interop] Better linker-friendly JavaObjectArray<T>.ValueMarshaler (#546) * dotnet/java-interop@4bd07e0 [vs-code] Recommend the zbecknell.t4-support extension (#545) * dotnet/java-interop@3b24a2c [Java.Interop] apply AggressiveInlining where profiler shows calls (#541)

dnfclas added the cla-already-signed label Mar 30, 2017

jonpryor added the do-not-merge PR should not be merged. label Mar 30, 2017

jonpryor force-pushed the jonp-vsix branch 3 times, most recently from c390259 to 3513aa6 Compare March 30, 2017 13:55

kzu reviewed Mar 30, 2017

View reviewed changes

jonpryor force-pushed the jonp-vsix branch 3 times, most recently from 4db94ef to 6b0dd12 Compare March 30, 2017 19:24

jonpryor removed the do-not-merge PR should not be merged. label Mar 30, 2017

kzu reviewed Mar 30, 2017

View reviewed changes

jonpryor force-pushed the jonp-vsix branch 2 times, most recently from e43508e to fa69fd4 Compare March 30, 2017 20:08

jonpryor added the do-not-merge PR should not be merged. label Mar 30, 2017

jonpryor force-pushed the jonp-vsix branch 4 times, most recently from 6836f6c to 69d1473 Compare April 3, 2017 17:15

kzu mentioned this pull request Apr 3, 2017

[msbuild] Preserve overriden values and only override on VS2017+ jonpryor/xamarin-android#1

Merged

jonpryor force-pushed the jonp-vsix branch 3 times, most recently from ff74feb to 7c57727 Compare April 4, 2017 02:03

jonpryor force-pushed the jonp-vsix branch from 7c57727 to 7cebd85 Compare April 4, 2017 02:07

jonpryor merged commit a08cd80 into dotnet:master Apr 5, 2017

jonpryor mentioned this pull request Dec 19, 2019

Bump to xamarin/Java.Interop/master@d61b329c #4059

Merged

github-actions bot locked and limited conversation to collaborators Feb 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[create-vsix] Create `.vsix` files #541

[create-vsix] Create `.vsix` files #541

jonpryor commented Mar 30, 2017 •

edited

Loading

jonpryor commented Mar 30, 2017

kzu Mar 30, 2017

kzu Mar 30, 2017

kzu Mar 30, 2017

kzu Mar 30, 2017

jonpryor Mar 30, 2017

kzu Mar 30, 2017

kzu Mar 30, 2017

[create-vsix] Create .vsix files #541

[create-vsix] Create .vsix files #541

Conversation

jonpryor commented Mar 30, 2017 • edited Loading

Footnotes

jonpryor commented Mar 30, 2017

kzu Mar 30, 2017

Choose a reason for hiding this comment

kzu Mar 30, 2017

Choose a reason for hiding this comment

kzu Mar 30, 2017

Choose a reason for hiding this comment

kzu Mar 30, 2017

Choose a reason for hiding this comment

jonpryor Mar 30, 2017

Choose a reason for hiding this comment

kzu Mar 30, 2017

Choose a reason for hiding this comment

kzu Mar 30, 2017

Choose a reason for hiding this comment

[create-vsix] Create `.vsix` files #541

[create-vsix] Create `.vsix` files #541

jonpryor commented Mar 30, 2017 •

edited

Loading